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Abstract 

Across K-12 education, there has been recent attention to the learning opportunities available to 
students in science, technology, engineering, and mathematics (STEM) learning. Early childhood 
education (ECE) has been excluded from this process. The scholarly literature contains good 
evidence for including science teaching and learning at the ECE level, however, little is currently 
known about how a STEM curriculum might best be implemented in the early years. Additionally, 
data collection tools specifically developed for ECE STEM are limited. In this paper, we outline 
the steps we have taken to design and validate a classroom observation protocol (COP) intended 
to capture aspects of STEM instruction in an early childhood setting and to help educators 
envision and adjust their own teaching practices. The detailed explanation of the development and 
validation of this data collection tool offers other educators and researchers a path for collecting 
evidence of STEM practices in their own classrooms and/or research environments. 

Keywords: STEM, early childhood education, Pre-Kindergarten education, Classroom 
Observation Protocol, data collection tools, development and validation 

Introduction 

This article reports a study of ways to examine the implementation of a science, technology, 
engineering and mathematics curriculum in the early years. The National Research Council of the 
United States of America (NRC, 2011) has identified three broad goals for STEM: an increase in 
advanced training and careers in STEM fields; an expansion of the STEM-capable workforce; and 
an increase in scientific literacy among the general public. The acronym STEM is sometimes used 
to refer to any one of the individual disciplines, sometimes to denote the integration of all four 
disciplines, and sometimes to mean a combination of two or more of the disciplines (National 
Academy of Engineering and National Research Council, 2014). Our perspective, shared by other 
early childhood specialists (c.f, Moomaw, 2013), is that an activity can be considered STEM if any 
two of the four disciplines are intentionally emphasized. 

One of the key goals of education is to establish an environment that supports an appreciation for 
lifelong learning; thus, an important question to raise is just what kinds of experiences within early 
childhood education (ECE) can foster such a disposition in children (Katz, 2010). There are 
suggestions that STEM at the early childhood level, if approached correctly, could offer 
opportunities for teachers to engage young children in activities that capitalize on students' 
interests, experiences, and prior knowledge (NRC, 2011). In this article we briefly make the case 
for including STEM in ECE, describe a project in which we are observing the implementation and 
effects of a Pre-Kindergarten (Pre-K) STEM curriculum, and then explain the development of a 
data collection tool that facilitated our observations. (Note that Pre-Kindergarten and Pre-K are 
terms used across North America to describe formal education programs for children who are four 
years old. Elsewhere, Pre-K education may be called Junior Kindergarten). This data collection 
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tool could be used to ultimately aid early childhood educators in regular assessment of teaching 
and learning. 

Setting the Scene for STEM in Early Childhood Education 

There is a large and comprehensive body of literature about what effective science instruction 
looks like, with a small section devoted to what effective science instruction might include in ECE 
(Leuchter, Saalbach, & Elardy, 2014). Suggestions within this literature are that science instruction 
at this level should address what children know and what they can learn, involve an inquiry 
approach, and provide appropriate scaffolding to foster conceptual understanding and reasoning 
(Furtak, Seidel, Iverson, & Briggs, 2012; Hardy, Jonen, Moller, & Stern, 2006; Leuchter et al., 2014; 
Tmndle & Sa^kes, 2012). Katz (2010) suggests that an appropriate science curriculum in the early 
years is one that encourages and motivates children to seek mastery of basic skills in the service of 
their intellectual pursuits. Rather than “delivering” education, educators are most likely to help 
children in their learning by “providing” experiences known to benefit young children. 
Additionally, Eshach and Fried (2005) argue that science is an important - and perhaps imperative 
- component of ECE as it builds upon students' innate interests in the natural world, can help 
develop positive attitudes towards the discipline, and can provide a foundation upon which further 
learning and understanding can be built. 

Overall there appears to be compelling evidence for including science in ECE, but what about 
including STEM? Even though early childhood experiences influence later academic success 
(Human Learning Early Partnership, 2009) and it is likely that quality ECE STEM experiences will 
lead to future academic success, attention to STEM education still tends to be focused on the 
secondary and middle school levels of the K-12 range, both in the literature and in the classroom. 
Nevertheless, there are several recent documents addressing connections between science and 
STEM education that have influenced our ongoing work in this area. First, the Position Statement of 
Early Childhood Education (National Science Teachers Association, NSTA, 2014), which focuses on 
children from three years through preschool, describes the capacity of children to engage in the 
process of science and develop conceptual understanding, the important role adults play in 
contributing to these early experiences, the varied opportunities that such experiences can take at 
this level and the time required to achieve such ends. The NSTA statement focuses on children’s 
predisposition to observe, explore, and discover the world they are surrounded by and affirms that 
learning science and engineering practices foster enjoyment, curiosity and lay the foundation for 
the progression of science learning thru their entire lives. Specifically, the position statement 
(NSTA, 2014) suggests that learning science and engineering practices in the early years (i) fosters 
children’s curiosity and enjoyment in exploring the world around them, and (ii) lays the foundation 
for science learning in K—12 settings. 

The second document we referenced was the Next Generation Science Standards (NGSS, Achieve, 
2013) which is based upon the Eramework for K-12 Science Education: Practices, Crosscutting Concepts and 
Core Ideas (NRC, 2012). The NGSS influenced our work in two ways. First, it clearly identifies and 
describes the major ideas of K-12 science education (i.e., that science is a series of cognitive, social, 
and physical practices of inquiry, which involves concepts that cut across science topics - such as 
patterns or change - as well as core content knowledge). Secondly, it integrates understanding the 
ideas of science with engagement in science practices and offers a prominent place to the ideas and 
practices of engineering. More specifically the NGSS outlines three dimensions in which science 
and STEM education can be integrated: (i) science and engineering practices: inquiry and problem 
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solving (design); (ii) cross-cutting concepts: key underlying ideas that are common to a number of 
topics; and (iii) core ideas: specific content and subject areas (Achieve, 2013). 

Both the NSTA (2014) position statement and the NGSS (Achieve, 2013) clearly emphasize 
aspects of engineering within and alongside science instruction. When this emphasis is considered, 
along with the categorization of mathematics as a foundational area at all education levels including 
early childhood (cf., BCME, 2008, OME, 2010), all that remains is the addition of technology to 
develop STEM at the early childhood level. This final leap is not too large when one considers that 
technology in this discussion is more than information communication technology and includes 
any tools that are used to solve problems. 

Our Objective/The Problem Space 

Drawing on the literature about science in ECE, and influenced by the NRC (2012), the NGSS 
(Achieve, 2013), and the NSTA (2014), we began an observational study in a Pre-K classroom in 
which two early childhood educators were implementing a STEM curriculum. These two educators 
had invited the first author to support them in their efforts and he subsequently asked the second 
author to join the group and create a formal research team. Together, we are examining the 
processes that these early childhood educators use in designing, implementing, and refining a 
STEM curriculum for students in Pre-K at a small independent all-girls school in British Columbia, 
with plans to expand to other primary grades and additional schools. The students and their 
parents are also participating in the study, enabling us to explore the perspectives of a range of 
stakeholder groups. 

The goal of our study is to investigate how STEM education might be effectively implemented in 
an early childhood learning environment, while highlighting benefits and challenges. The study 
includes an action research perspective, with the participating educators reflecting on their current 
STEM curriculum and acting on personally meaningful questions. 

Key objectives include: 

• collecting data within a Pre-K learning environments to better understand the design, delivery, 
and challenges of STEM education at this level; 

• determining areas to target for improvements in Pre-K STEM curriculum; and 

• developing suggestions for implementing STEM programming in ECE. 

As is the case in all research, appropriate data collection tools are a necessary component of this 
study. However, although observation protocols exist for STEM teaching, such as the Electronic 
Quality of Inquiry Protocol (EQUIP, Marshall, Horton, & White, 2009) and for ECE classroom 
environments, such as the Classroom Assessment Scoring System (CLASS, Le Paro, Pianta, & 
Stuhlman, 2004) and there is literature available on how to go about selecting and/or using such 
tools (e.g.. Institute of Educational Services, 2009; Le Paro et al., 2004; Marshall et al., 2009), there 
was nothing we could find in the literature that focused on the overlap between STEM learning 
and ECE learning environments. Thus, we decided to draw on literature focused on STEM with 
older students (e.g., Brown, 2012), early childhood education (e.g., Le Paro, Pianta, & Stuhlman, 
2004), or early childhood science (e.g., Trundle & Sa^kes, 2012) to design and validate a classroom 
observation protocol (COP) that we could use in our research project, and that would also be 
useful as an assessment tool for ECE teachers. 
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Development of the ECE STEM COP 

To provide the foundation for a Classroom Observation Protocol (COP) for STEM in ECE, and 
develop a tool that could provide snapshot evidence of classroom interactions within ECE STEM 
education, we reviewed relevant standards and position statements: the Framework forK-12 Science 
Education: Practices, Crosscutting Concepts and Core Ideas (NRC, 2012), the NGSS (Achieve, 2013), and 
the Position Statement of Early Childhood Education (NSTA, 2014). Then, since effective practices for 
STEM are closely related to effective practices for education in general (e.g., NRC, 2011), we 
analyzed two provincial government framework documents on quality ECE programs. We selected 
items from British Columbia and Ontario, since these are the provinces where our respective 
universities are located, and they are also the provinces containing two of the three largest urban 
regions (i.e., Toronto and Vancouver) in the country (Statistics Canada, 2013). 

The first framework we drew on was the British Columbia Early Earning Framework (ELF, British 
Columbia Ministry of Education, BCME, 2008). The EEF is a guide for early childhood educators 
(among others) who are providing learning experiences for children from birth to kindergarten. 
This document outlines early learning as young children’s physical, intellectual, emotional, social, 
and creative capacities and emphasizes the natural curiosity that children are born with, the role 
that others in their lives can take in supporting learning, and the role that early learning plays in 
establishing a foundation for lifelong learning. The EEF (BCME, 2008) identifies key areas for 
learning such as (i) well-being and belonging (explore their own strategies for learning), (ii) 
exploration and creativity (explore the world using all their senses; build, create, and design; 
actively explore, think and reason; identify and try possible solutions to problems; develop a sense 
of wonder for natural environments), (iii) languages and literacies (develop the capacity to 
communicate; use numbers, measurement, and form), and (iv) social responsibility and diversity 
(understand how their actions may affect nature). 

The second government framework, Ontario's Early Earning-Kindergarten Program (EEK, Ontario 
Ministry of Education, OME, 2010), describes a strong foundation for learning in the early years 
that promotes the physical, emotional, and cognitive development of all children. The document is 
based upon six principles: (i) early learning is foundational for lifelong learning, (ii) family and 
community partnerships are helpful in meeting young children's learning needs, (iii) respect for 
diversity, equity, and inclusion are essential, (iv) planned curriculum is necessary, (v) play capitalizes 
upon early learners natural inclinations towards curiosity, and (vi) knowledgeable educators are 
essential. More detailed than the EEF (BCME, 2008), the ELK (OME, 2010) is broken into 
learning areas and program expectations around language, mathematics, science and technology, 
inquiry and forms of play. 

Comparing these two ECE frameworks, we identified five aspects of effective early learning 
environments that were common to both, that would be relevant for STEM education, and that 
would be integrated into the COP. These aspects and their description in the two documents are 
shown in Table 1. 
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Table 1 Aspects of Early Childhood Education that are Relevant for STEM Education 


Aspect 

Questioning 

Exploring and 
Observing 


Developing S kill s 
and Processes 


Communicating 


Playing 


ELF { BCME, 2008) 

• ask questions, explore, and make 
discoveries 

• actively explore, think and reason 

• explore the world using all of their 
senses 

• build, create, and design using 
different materials and techniques 

• use numbers, measurement, and 
form 

• identify and try possible solutions 
to problems 

• develop diverse language abilities 
and communicates with others 

• play is vital to children’s healthy 
development and learning 


ELK{ OME, 2010) 

• raise questions about objects and 
events around them 

• explore objects and events and 
observe the results of explorations 

• make observations, using all of 
their senses, and generate questions 

• gather, compare, sort, classify, 
order, interpret, describe 
observable characteristics and 
properties, 

• notice patterns, and draw 
conclusions, using a variety of 
simple tools and materials 

• work individually and with others, 
and share and discuss ideas through 
talking, listening, writing 

• content is learned through play, 
investigation, and intentional 
teaching 


These five aspects formed the foundation for our ECE STEM COP. The NGSS (Achieve, 2013), 
the ELF (BCME, 2008), and the ELK (OME, 2010) contributed to our development of dimensions 
for each aspect and of indicators that would provide evidence of such dimensions as they were 
enacted in an ECE classroom. We also drew heavily upon the EQUIP protocol (Marshall et al., 
2009) and utilized materials from Early Childhood Hands-On Science (ECHOS 1 *), a science 
curriculum that emphasizes fundamental science concepts and science process skills (Patricia and 
Phillip Frost Museum of Science, PPFMS, 2014). Additionally, we referred to the provincial 
science curriculum documents from British Columbia and Ontario. For example, for the Play 
aspect, we drew directly from both the ELF (BCME, 2008) and ELK (OME, 2010) documents, 
using key phrases in the COP. Phrases such as communicate by talking, listening and writing explore 
recognise, describe and create patterns; use technologicalproblem solving and discussing ideas, sharingfinding were 
all integrated directly into the COP. 

Using these documents and taking into account the realities of the classroom environment, we 
designed a preliminary COP for STEM learning at the early childhood level. The process of 
deciding upon aspects, dimensions, and indicators was as follows: 

• The Questioning aspect was most influenced by EQUIP (Marshall et al., 2009). 

• Exploring became part of the Play aspect, which was informed by the ELF 
(BCME, 2008) and the ELK (OME, 2010). 
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• Observing was subsumed by the Process Skills aspect which was in turn heavily 
influenced by ECHOS (PPFMS, 2014) and validated by comparison with the two 
provincial curriculum documents. 

• Communicating was also merged with Process Skills. 

• A fourth aspect, Scientific and Engineering Practices, was added to the COP and 
appropriate dimensions for this aspect were adapted from the K-2 performance 
expectations of the NGSS (Achieve, 2013). We wanted to differentiate between 
process skills and science and engineering practices so that the COP could be used 
for both science and STEM observations. 

At this stage, the COP consisted of four aspects, 16 dimensions, and three to five indicators for 
each dimension. For consistency and ease of use, we condensed the number of indicators so that 
there were three for each dimension. For example, the Predicting dimension in the Process Skills 
aspect was drawn from ECHOS (PPFMS, 2014) and initially consisted of five indicators: verbalising 
thinking, recognises and extends patterns, makes simple predictions, makes predictions based upon observations, 
and uses estimates to make quantitative predictions. Because we already had an Observing dimension, we 
reduced the number of indicators to four: verbalising thinking, recognises and extends patterns, makes 
simple predictions, and uses estimates to make quantitative predictions. Then, since the concept of qualitative 
and quantitative were already addressed in the Math and Computational Thinking dimension, and 
Communicating subsumed verbalising thinking, we further reduced the number of indicators to two: 
recognising and extending patterns, and making simple predictions. The third indicator for the Predicting 
dimension, comparing predictions to what actually occurred, came from the NGSS (Achieve, 2013). 

We piloted our initial COP and refined it for clarity, added items that classroom use suggested 
should be included, and removed indicators that were duplicated across dimensions. After piloting, 
we added a code number to each indicator to enhance the usability of the COP, making 
observations more efficient because only the code would need to be recorded, rather than a 
description of the indictor. Ease of use was essential because our expectations of the COP were 
that it could and would be used in real time (i.e., while instruction was taking place). 

The current COP, which is still in a trial phase and should not be used without the written 
permission of the authors, addresses four aspects (e.g., scientific and engineering practices) with 
each aspect being composed of two or more dimensions (e.g., argument). Additionally, each 
dimension has three separate indicators or types of behaviours that likely suggest the dimension is 
present (e.g., indicating agreement or disagreement based on evidence). The Process Skills aspect 
of the COP for ECE classrooms is presented in Table 2. 
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Table 2 The Process Skills Aspect of the ECE STEM COP 


Asnect r 

“ Process Skills 


Dimension 


Indicators 


Observing 

Using senses to identify 
properties of objects 

Using tools to observe 
objects and events 

Using measurement tools 
to record observations 

Describing 

Describing key 
attributes of objects 

Creating drawings or 
models depicting 

nbiertt 

Describing changes in 
objects 

Categorizing 

Noticing similarities 
and differences 

Sorting objects into 
groups using one or 
more attribute 

Establishing and 
justifying sorting criteria 

Predicting 

Recognizing and 
extending patterns 

Making simple 
predictions 

Comparing predictions to 
what actually occurred 

Communicating 

Communicating 
information or design 
ideas +/or solutions in 
oral + /or written forms 

Sharing, listening, and 
discussing ideas 

Communicating results 
and findings 


Our classroom observation protocol is intended to aid in the precision and accuracy of capturing 
early childhood classroom behaviours, and therefore it needs to be practical for use by early 
childhood educators, researchers, and research assistants (RAs). We wanted to create a COP that 
requires minimal 'training' so that it is user friendly and classroom ready; teachers will be able to 
use it without the extensive training that some other COPs require. For example, CLASS requires 
intensive training before a reliability test is taken by data collectors (Le Paro et al., 2004). The initial 
steps of developing this COP involved both of the researchers over a series of sessions in the Pre- 
K classroom creating, piloting, editing, and re-testing the tool. The early childhood educators were 
not involved in the creation or early revisions of the COP and only began to work with it after we 
had finalized the current version, because we wanted to provide them with a tool that we deemed 
workable without overwhelming them with the many changes that occurred in the initial stages of 
development. 

We asked the early childhood educators to use the COP along with a parallel lesson planning 
template (see Figure 1) that highlighted the dimensions and indicators the lessons were intended to 
demonstrate. We also asked the educators to reflect on their lesson plans using the parallel 
template. There was an initial resistance to the use of the COP from one educator, because her 
preferred approach to collecting classroom data was anecdotal running records. She also expressed 
a sense of discomfort arising from 'forcing' children into the actions and behaviours described in 
the COP. However, after we emphasised that the COP was tied directly to both the lesson 
planning and lesson reflection templates, and reinforced that this tool was merely observational, 
with its use supplementing rather than replacing anecdotal recording, she was willing to try out the 
COP and the planning template. 
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Date: 

Topic # and Title: 

Pre-Teaching Reflection 

Intended Dimensions of emphasis-. 

Possible challenges-. 

Post-Teaching Reflection 

Flow the lesson went 

Evidence-. 

Evidence of (Indicator) during STEM instruction 

Actual Dimension enacted 









How the lesson successfully addressed the intended Dimensions-. 

Changes to make to lesson plan and instruction (e.g., to build on unanticipated strengths or address noted 
challenges): 

The best thing (just one!) that I saw or thought about this lesson-. 


Figure /. The ECE pre and post lesson reflection template. 


After this initial reticence, the feedback from the early educators on the use of the COP (combined 
with the lesson and reflection templates) was positive. Both educators were very happy with the 
linkage between the COP and the lesson reflection template as the benefits of this connection (i.e., 
reduced time in planning lessons and completing the lesson reflection template, ability to quickly 
capture classroom behaviours) became immediately apparent to them. In fact, when asked to talk 
about the COP, the educators commented that “It was extremely helpful in [our] planning and 
assessment of the activities [that we were doing in the classroom]. It was rich and not cumbersome 
to use once we became comfortable with it. Numbers for identification of dimensions and aspects 
was useful.” They also commented that “It helped us identify and name the STEM skill that the 
children were exhibiting and those attributes we were looking for. It helped us to be more 
intentional with our planning. It gave us a road map for assessment.” 

In an effort to increase the precision and accuracy of the use of the COP in the classroom for early 
childhood educators to plan and evaluate their lessons, as well as enable use by other researchers, 
we developed descriptors and exemplars of what these indicators might look like in practice. The 
ease of use of the COP was enhanced when all members of the research team were provided with 
brief descriptors and examplars for each indicator. An example of the flow from aspect to 
exemplar for each of the four aspects is shown in Table 3. 
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Table 3 Sample Descriptors and Exemplars for Indicators on the ECE STEM COP 


Aspect 

Dimension 

Indicator 

Descriptor and Exemplar 

Questioning 

Characteristics 
and Nature 

Q-l Range of 
purposes: remember, 
understand, apply, 
analyze, create, 
evaluate 

The emphasis is on the function of the 
question and the kind of response that 
the question is intended to encourage, 
i.e., in a particular context a variety of 
question types are asked, e.g., ‘Who 
remembers...?”, Why do you think...? 

Play 

Pretend 

P-3 Maldng mental 
representations 

using imagination; using one item in the 
place of another, e.g., while building with 
blocks a child comments "I am making a 
house for my friends" 

Process 

Skills 

Categorizing 

PS-8 Sorting objects 
into groups using one 
or more attribute 

Buttons can be sorted by shape, but also 
by color, number of holes, material, etc. 

NGSS 

Scientific 

and 

Engineering 

Practices 

Argument 

SEP-14 Indicating 
agreement or 
disagreement based on 
evidence 

A student agrees or disagrees with a claim 
and explicitly refers to relevant previous 
experiences ("I agree that this object will 
float because it is squishy and full of 
holes like a sponge, and a sponge 
floated") 


Our next step was a validation process that addressed the validity and reliability of the COP. We 
needed to identify which dimensions and indicators required revisions and which descriptors 
needed rewriting. Through this process we would also be able to determine whether the training 
process was sufficient for future scale-up. 

Validity and Reliability 

The first basic property of empirical measurements is validity (Watkins & Pacheco, 2000). The 
criterion-related validity of the ECE STEM COP (i.e., connections between the external 
behaviours and the measuring instrument itself) can be assessed by how well the COP is linked to 
the numerous policy, government publication and curricular documents referenced in its creation 
(Carmines & Zeller, 1979), in particular the NGSS (Achieve, 2013), the ELF (BCME, 2008),and 
the ELK (OME, 2010). Our arguments for criterion-related validity, and by extension support for 
the appropriate interpretations from the results of this tool, rest upon the details provided in this 
article for the design of the COP. 

With the validity of the COP established, the second basic property of empirical measurement 
(reliability) had to be determined (Watkins & Pacheco, 2000). In this instance, inter-rater reliability 
(IRR) was the appropriate calculation. However, there is little consensus about the best statistical 
measures for determining IRR for COPs despite agreement that reliability is critical for 
development and that it is demanded by funding agencies (Rui, 2009). Faced with this lack of 
consensus, we chose Cohen's kappa coefficient (x) as our measure of IRR because it compares the 
observed agreement between two raters with the agreement expected by chance alone (Hallgren, 
2012) which made the calculation particularly appropriate given our validation process, which 
involved coding of video clips by multiple members of the research team. 


ISSN 1838-0689 online 

Copyright © 2010 Monash University 

www.education.monash.edu.au/irecejournal/ 













International Research in Early Childhood Education 
Vol. 6, No. 1, 2015, page 33 


To calculate the IRR we video recorded two STEM lessons in the Pre-K classroom and then 
segmented the recordings into nine shorter clips. The nine clips followed natural breaks and 
transitions in the lessons and were approximately five minutes in length. We assigned six different 
video clips to each member of the research team, ensuring that each clip was coded four times, as 
shown in Table 4. Additionally, each rater coded the same clip as each other rater at least three 
times to allow pairways comparisons. For example, both Researcher 1 and Researcher 2 coded 
Clips 1,2, and 9. 


Table 4 Assignment of Video Clips for Calculating the Reliability of the ECE STEM COP 


Video Clip 

Rater 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Researcher 1 

X 

X 

X 


X 


X 


X 

Researcher 2 

X 

X 


X 


X 


X 

X 

ECE Teacher 1 


X 


X 

X 

X 


X 

X 

ECE Teacher 2 

X 


X 

X 


X 

X 

X 


Research Assistant 1 


X 

X 


X 


X 

X 

X 

Research Assistant 2 

X 


X 

X 

X 

X 

X 




We examined the agreement between pairs of raters on each video clip. Our approach was to 
compare the observer reliability on all the 48 indicators in the COP and to calculate k values for 
each pair. For example, for each occurrence of an indicator raters could (i) choose the same 
indicator, (ii) choose no indicators, (iii) rater A could choose an indicator when rater B did not, or 
(iv) rater B could choose an indicator when rater A did not. To facilitate the pairways comparisons, 
we broke the video clips down into 10 second segments, determined the instances of rater 
agreement for each segment, and then totalled the instances. For example, Researcher 1 and RA 2 
had the following results: (i) both raters indicated the same code 17 times, (ii) both raters did not 
indicate 4 times, (iii) rater A indicated 1 time when rate B did not, and (iv) rater B indicated 6 times 
when rater A did not. Using these data we will be able to calculate the percentage agreement as 
well as x between all sets of raters. Initial results of the comparison are provided in Table 4. 

Table 4 Partial Output for COP Rater Tasks Selection for Individual Videos 
Video Raters # Total Tasks % Agreement Kappa (x) 

Clip Agreement 


1 

Researcher 1 and 

Research Assistant 2 

21 

28 

0.75 

0.40 

4 

Researcher 2 and 

Research Assistant 2 

53 

68 

0.78 

0.41 

5 

Researcher 1 and 

Research Assistant 2 

48 

60 

0.80 

0.44 

9 

Researcher 1 and 

Researcher 2 

14 

18 

0.78 

0.41 
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Despite the fact that x has not been calculated for the remainder of the video coding, a few general 
statements on the reliability of the COP can be made. For example, a rough estimate of the 
percentage of agreement (i.e., expressed as the number of agreements in observations divided by 
the total number of observations) was calculated for two raters across the first video at 75%. 
Additionally, x for those same set of raters was .40. Interpretation of x is straightforward as it takes 
the form of the common correlational coefficient (i.e., bounded range of +1 to -1). Values of less 
than .40 are poor, values of .40 to .60 suggest fair agreement, values of .60 to .75 represent good 
agreement, and values greater than .75 indicate excellent agreement (Watkins & Pacheco, 2000). All 
calculations for % agreement and x will be performed as above, once all raters have completed 
coding their video clips. 

What is perhaps of more interest than how often raters’ codes were in agreement is the areas 
where differences in coding were observed. The areas appearing problematic (i.e., perhaps 
requiring revision of the COP or more attention to training) were within the aspects of 
Questioning (e.g., range of purposes versus level of question) and Process Skills (e.g., simple 
predictions versus communicating information) and between indicators in the aspects of Play and 
Process Skills (e.g., making observations using all senses versus noticing similarities and 
differences). The analysis of smaller segments is helping to aid in our identification of problematic 
indicators, for example, areas of overlap both within and between aspects of the COP. Our plans 
to refine the ECE STEM COP include addressing all areas in which IRR is low, which will result in 
a more useful and reliable tool for both teachers and researchers in ECE. 

Discussion and Next Steps 

This paper presents the initial phases of a study that - through the combined efforts of two 
researchers and two early childhood educators, not to mention a classroom of young learners and 
their parents and guardians - is exploring how STEM principles might be effectively implemented 
in an early childhood learning environment. The questions that guided us in the initial phases of 
the study included (i) What are the aspects of effective ECE STEM instruction? and (ii) Flow can 
we gather evidence of those aspects? In an effort to gather data towards uncovering answers to 
these questions, we designed and are currently validating a COP that can be used to collect 
information about the nature of STEM instruction within an ECE classroom. The COP has been 
piloted for use in conjunction with a lesson planning and reflection template to offer a clearer view 
of the STEM teaching and learning occurring in the Pre-K classroom and to provide feedback for 
the early childhood educators on teacher and student interactions during STEM lessons. The 
reflection template (i.e., Figure 1) is already in place and being used by the early childhood 
educators, the next step is to implement it as a planning tool in their practice. 

STEM is at the forefront of current educational debate and thus exploring STEM programs in 
ECE is both a timely and potentially fruitful pursuit. There are a number of uses for the COP and 
the data that it can generate. For example, it is suggested that students at the ECE level need 
STEM experiences that will help them develop their intellectual capabilities (cfi, NSTA, 2014). The 
COP could assist early childhood teachers to plan STEM experiences and ensure that such 
experiences are in fact taking place in their classrooms, particularly when combined with the lesson 
planning and reflecting template. The COP can also be utilized to identify STEM indicators that 
are not evident throughout lessons and activities, and thus providing teachers a non-threatening 
tool for teachers to analyze and adjust their planning and instruction to promote these indicators. 
This COP, although still in development, enables us to broadly capture the behaviours of ECE 
students in the context of STEM education. The aspects, dimensions, and indicators of the COP 
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were adapted from a number of public, policy, and governmental documents associated with 
science education, STEM education, and ECE. We designed the COP based upon documents that 
(i) were recent within the literature around science and STEM education, (ii) were relevant to the 
geographical location of the study, and (iii) offered guidelines for assessing the ECE environment. 
The aim is that the COP will be a viable research and teaching tool, aiding researchers in collecting 
data and helping early childhood educators to plan, implement, and reflect upon their STEM 
practices. There is an obvious economic argument for effective ECE STEM programming as 
nations seek to compete in the knowledge economy; however, the significance of effective 
programming should go beyond economics. STEM education is not just for those students who 
will pursue post-secondary education or careers in STEM-related fields. A population with a 
foundation in STEM will be better prepared to face the challenges of a science and technology 
driven society (NRC, 2011). 
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